System and Method for Converting Input from Alternate Input Devices

ABSTRACT

An apparatus for delivering alternate user input between an alternate input device and an output device, the output device configured to receive input from a conventional input device, the output device not configured to receive input from the alternate input device, the apparatus including an input interconversion processor that receives the alternate user input from the alternate input device, a processing pipeline that converts the alternate user input to a conventional user input of a type normally received by the output device from the conventional input device, and an output port that transmits the conventional user input.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Patent Application Ser. No.62/141,620, filed Apr. 1, 2015, the contents of which are herebyincluded by reference in their entirety.

BACKGROUND

A wide range of electronic systems, including for example personalcomputers, appliances, vehicles and industrial manipulators, can becontrolled by user input in some form. These systems are often “tightlycoupled” to a particular set of user interface devices (e.g. mouse &keyboard, button panel, steering wheel & foot pedals, joysticks)—thatis, these systems are built to be used with those specific devices ortypes of devices, and do not naturally offer support for other devices.In some cases the supported user interfaces may be undesirable orprohibitive for a particular user or use case, and in such cases itwould be beneficial to customize users' interactions with the system.For example, someone with a physical disability affecting hand functionmay not be able to effectively use a standard game console controller toplay a video game, whereas they may be capable of playing the video gameusing other controls, such as, for example, a combination of headcontrols and foot controls, or reconfigured existing controls, if onlyit were possible to control a gaming console using such inputs or insuch a method. Many electronic systems are designed to support only asmall fraction of existing user interface devices and provide limitedconfiguration options to configure the use of these devices. In manycases, it is not straightforward to reconfigure a system to use userinterface devices other than those it was designed to support, and thususers are left with limited options in cases where the available userinterface devices are unsuitable for various reasons. To address thisproblem, we have invented a modular, reconfigurable, interconvertinguser interface routing system. This system allows users to useelectronic systems in ways adapted to their needs and preferences ratherthan users needing to adapt to the natural configuration of the systems,by configuring our system to use their preferred user interface devicesto control a system they desire to control in a manner that suits theirneeds or desires.

The system enables such reconfigurable user interface compatibility, aswell as a number of further enhancements such as:

-   -   The system can process, combine, and transform input signals to        create new behaviors not designed into either the user interface        devices being used or the system being controlled.    -   The system may provide a live-updating graphical interface,        allowing users to rapidly configure and reconfigure their        controls, thus reducing the barriers to optimizing their        controls or switching between use cases.    -   The system can log user inputs and intermediate signals,        enabling analysis and reporting of system usage (applications        range from safety/prevention to patient-tracking during physical        therapy).    -   The system enables multiple users to control the system        collaboratively using multiple user interface devices (e.g., for        training or assistance).    -   The system can modulate user controls according to measurements        or inputs (e.g., as part of a physical therapy regimen, decrease        resistance as the user fatigues; as part of a training exercise,        decrease assistance as user improves).

SUMMARY OF THE INVENTION

An apparatus for delivering alternate user input between an alternateinput device and an output device, the output device configured toreceive input from a conventional input device, the output device notconfigured to receive input from the alternate input device, theapparatus including an input interconversion processor that receives thealternate user input from the alternate input device, a processingpipeline that converts the alternate user input to a conventional userinput of a type normally received by the output device from theconventional input device, and an output port that transmits theconventional user input.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the high-level architecture of the system.

FIG. 2 shows an example architecture of a standalone implementation ofthe system.

FIG. 3 shows an example architecture of an OS kernel implementation ofthe system.

FIG. 4 shows an example architecture of a SoM/SoC implementation of thesystem.

FIG. 5 shows an example of the system directly interfacing with inputdevices and output systems according to an aspect of the system of thepresent disclosure.

FIG. 6 shows an example of the system interfacing with input devices andoutput systems via an intermediary protocol converter according to anaspect of the system of the present disclosure.

FIG. 7 shows a graph-based architecture of the signal processingpipeline according to an aspect of the system of the present disclosure.

FIG. 8 shows the architecture of an aspect of the system of the presentdisclosure.

FIG. 9 shows the operation of the main application component of anaspect of the system of the present disclosure.

FIG. 10 shows the functionality of a signal processing node according toan aspect of the system of the present disclosure.

FIG. 11 shows a signal processing node with a “port array” according toan aspect of the system of the present disclosure.

FIG. 12 shows a series of example signal processing nodes according toan aspect of the system of the present disclosure.

FIG. 13 shows a series of example stateful signal processing nodesaccording to an aspect of the system of the present disclosure.

FIG. 14 shows an example system configuration according to an aspect ofthe system of the present disclosure.

FIG. 15 shows a pane of an example GUI for system configurationaccording to an aspect of the system of the present disclosure.

DETAILED DESCRIPTION

Turning now to FIG. 1 , a key innovation of the present disclosure is toinsert a modular, reconfigurable signal processing pipeline 201 betweenuser interface devices 101 and the systems desired to be controlledusing them 102. The processing pipeline 201 includes an API 202.

The system accepts signals from one or more user interface devices 101,sends them through the signal processing pipeline 201, and sends thesignals output by the signal processing pipeline to one or more outputsystems to be controlled 102. The topology of the pipeline 201 and thetransformations applied are configurable or editable via an API 202.

Another key innovation of one aspect of the present disclosure is thatit combines a user-friendly configuration interface 701 with the signalprocessing pipeline 201 between user interface devices 101 and thesystems desired to be controlled using them.

By combining widely-compatible interconversion with the highlyconfigurable signal processing pipeline and user-friendly interface, thesystem enables unprecedented user interface device adaptability.

Architectures

The system can be implemented in many configurations, as heterogenous orhomogenous systems. It may be that each component shown in FIG. 1 isimplemented as separate software and/or hardware systems, with somemeans of interprocess, network, or other inter-system communication; itmay be that one or more components are implemented as part of the samehardware or software system; it may be that one or more components areimplemented using more than one hardware or software system.

In one aspect, with an example architecture shown in FIG. 2 , the systemmay be implemented as a “standalone unit,” 901 which is a singlephysical entity consisting of physical and/or wireless interfaces 104for user interface devices 103 and output systems 108, and one or moreprocessors and/or associated electronics for implementing I/Ointerconversion, the processing pipeline, and the API. A standalone unitmay be implemented in many configurations; for instance, theinterconversion and pipeline may for example be implemented using anFPGA, or may for example be implemented using a processor with dedicatedelectronics for device communication. In this “standalone” case, thesystem may for instance be used as a “plug-and-play” device used to makeoff-the-shelf user interface devices compatible with off-the-shelfoutput systems. A standalone system may include other features such asproviding power to input devices, incorporating optoisolation or othermeans of shielding from interference from input devices, or secondaryfeatures not necessarily related to the primary function of the system,such as a media player.

In another aspect, with an example architecture shown in FIG. 3 , thesystem may also be implemented as part of an operating system, forinstance as part of a Linux kernel 902 providing services to the OS 906.In this case all interconversion may already be handled by the OS 105and underlying device hardware 104, and the system would contain a“kernel space” processing pipeline 203 and would expose configurationhooks 803 to system users and applications, which could be used toconfigure to transform user interface device data at a kernel levelbefore it is exposed to “user space” applications 106.

In another aspect, the system may also be implemented as part of anapplication such as a game, as a means of user configuration of theapplication. In this case all interconversion may already be handled bythe OS and underlying device hardware, and the system would consist of aprocessing pipeline within the application that would be configurable bythe application user, and used to transform user interface device datafor application use without the need of kernel level configuration orthe use of a standalone adapter.

In another aspect, with an example architecture shown in FIG. 4 , thesystem may be implemented as “system on a chip” (SoC) or “system on amodule” (SoM) 903. In this case the system is on a single chip (orchipset) or circuit board module 903 with physical and/or wirelessinterfaces 104 to user interface devices 103 and output systems 108, oneor more processors and/or associated electronics for implementing I/Ointerconversion 105, the processing pipeline 203, and the API 803. As aSoC/SoM 903, the system may for instance be used for integration intoanother system (such as a power wheelchair or a cable box), for instanceto make that system compatible with off-the-shelf user interface devicesnot designed for use with that system.

Input Devices and Output Systems

Input devices may include, for example, human interface device (HID)devices, game controllers, switches, knobs, sliders, joysticks, or otherelectromechanical devices, motion capture systems, voice input or voicerecognition systems, wearable sensors or sensors that may be affixed tothe body, robotic devices, rehabilitation equipment,electroencephalography (EEG) systems, electrocardiography (EKG) devices,electromyography (EMG) devices, or other devices or systems thatgenerate one or more electronic signals, or that may be modified togenerate one or more electronic signals (for example by adding one ormore sensors).

Output devices may include, for example, computer applications, videogames, virtual reality systems, virtual interactive displays, industrialmanipulators, heavy machinery, automobiles, aircraft, or other systemsor devices controllable in part or in whole by electronic signals.

Signal Interconversion

Input and output device data is marshalled into and out of the systemvia hardware or software components (physical ports, dedicatedprocessors, device drivers, etc.) that accommodate the range of inputdevices and output systems supported by a particular implementation.Aspects of the system may have dedicated physical hardware for devicecommunication (such as USB ports, bluetooth ports, switch closure ports,custom ports etc.), pre-installed electronic hardware for devicecommunication (such as USB PHY, custom, etc.), and pre-installed devicedrivers (such as USB HID drivers, bluetooth HID drivers, custom drivers,etc.). The system may also support expansion of device compatibilitythrough various methods, such as daisy-chaining via existing deviceports, physical insertion of port expanders, replacement of electroniccomponents, installation of device drivers, and reprogramming of FPGAdrivers. Such expansion methods may also support hot-swapping while thesystem is running using various hot-swapping implementation methodsreadily available to those skilled in the art.

In various aspects different connected input devices and output devicesmay use different means of transmission (e.g., raw analog signals,hardwired digital protocols such as universal serial bus (USB), wirelessdigital protocols such as WiFi Direct) to communicate their data. Asshown in FIG. 5 , an aspect of the system may communicate with an inputdevice 103 or output system 108 by natively implementing that device orsystem's communication protocol for a port (defined as a physical orwireless connection) 104 that may be connected directly to a port on theinput device 103 or output system 108. Communications protocols, such asserial peripheral interface (SPI), universal serial bus (USB),controller area network (CAN), and other communications protocols, maybe implemented using any one or any combination of a digital processor,an algorithm implementing aspects of a published or unpublishedcommunications protocol system, purpose-designed electronic circuits andcomponents, or other electronic means.

As shown in FIG. 6 , an aspect of the system may also communicate withan input device 103 or output system 108 via a separate protocolconversion device 107 that may connect between that input device oroutput system (physical connection or wireless connection) and anavailable port (physical connection or wireless connection) of thesystem 104. The protocol conversion device 107 may also be configured tointerconvert the communication method of the input device 103 or outputsystem 108 and the communication method of the connected port on thesystem 104. For example, consider an input device that uses a USB 2.0protocol to communicate. An aspect of the system may for example includea USB 2.0 port to which the input device may connect via its own USB 2.0port, and communicate using a combination of hardware designed toprocess USB 2.0 communication signals and one or more processing unitsprogrammed to understand and use the processed USB 2.0 communicationsignals.

As another example, an aspect of the system may include an SPI port towhich an external SPI to USB interconverter may be connected, which maybe connected also to the input device. In this arrangement, the systemport may communicate using the SPI protocol while the input devicecommunicates using the USB 2.0 protocol, and the SPI to USBinterconverter allows communication emitted by the system as SPI signalsto be received by the input device as USB 2.0 signals, and communicationemitted as USB 2.0 signals to be received as SPI signals. In thismanner, input device ports may communicate with system ports, providedthat either the system can connect to the input device port andimplement its communication protocol, or that an interconverter is usedthat can facilitate connection of the system port and the input deviceport and communications between them. This approach applies both tocurrently available protocols and protocol interconverters, to protocolsfor which no interconverters currently exist (but which may bedeveloped), and to protocols and protocol interconverters that may bedeveloped. An aspect of the system also may include one or moreexpansion ports and analog and digital input boards. Expansion ports mayaccommodate more devices than there are physical ports for on thesystem. Analog input boards may have any number of input channels, eachdirectly measuring transmitted voltages, currents, or other signals fromany connected electronics. Digital input boards may have any number ofinput channels, each directly measuring transmitted boolean signals fromany connected electronics.

Signal Processing Pipeline

Routing and Transformations

The purpose of the signal processing pipeline is to allow the system toroute and transform signals received and interconverted from one or moreinput devices singly or in various combinations to produce signalsuseful for interconversion and output to control output systems, formeasuring characteristics of a user's interaction with the inputdevices, for evaluating characteristics of a user's ability, and forother uses by and within the system. Any component of the system havingone or more processing units, such as the main signal processingpipeline or an auxiliary processing device, may transform the signalsvia one or more transformations, functions, and methods describedherein, or other transformations, functions, or methods which may beuseful, to one or more intermediate signals before transformation to asignal used to send output to an output system. The system may routesignals from one or more particular input devices, via transformations,multiplexing, or various intermediate forms, to one or more particularoutput devices, according to the configuration of the system by a useror other means. Input devices may be represented as one or more analogor digital “channels,” with each channel corresponding to one or moredegrees of freedom, manipulable states, or other aspects of an inputdevice which a user may affect, and one or more associated signals. Thesystem may buffer, store, multiplex, or convert signals between andamongst various representations and formats. The system may representsignal values digitally, in analog, or in other electronic formatsuseful in representing real numbers, boolean numbers, discrete numbers,and other values. To accomplish this mapping from input signals tooutput signals, the system or a connected computing device may includeone or more processing units, and executable code that is configured tocause the processor to implement all or part of this functionality. Forexample, consider a user interface device comprising two separatelever-like user interfaces, and that is configured to produce signalscorresponding to a user's manipulation of the interfaces. The signalsmay each be represented as a channel, and the system may transform thesignals to a normalized representation (see transformations describedherein) and then send them via multiplexing to the horizontal andvertical axes of the right joystick included in a game system controllerthat may be attached to the output, such that output signals are sent tothe controller or output system that emulate the native signals of thecontroller.

Architecture

In one aspect, as shown in FIG. 7 , the signal processing pipeline maybe implemented as a data flow graph with executable nodes. “Graph” 205means a set of “nodes” 301 and connections (or “edges” 206) betweennodes. “Dataflow graph” means that certain nodes may be executed inresponse to data being available at their input ports. “Executablenodes” are connection points in a graph that have code that executeswhen data is available at their input ports, and may send data (forexample to other connected nodes) via their output ports. Nodes mayexecute within the same process or may execute in separate process onone or more computing systems. Data can likewise pass from output portsto input ports via direct function calls in the same process,interprocess communication (IPC), networked communication such as TCP orother means of communication. Data enters the pipeline via “sourcenodes” 302 that have one or more inputs accessible by one or moresources outside of the graph and exits via “sink nodes” 303 that haveone or more outputs available to one or more recipients outside of thegraph.

This aspect advantageously allows the user a great deal of flexibility,customizability and clarity in routing and transforming input signalsfor control of output systems. In other aspects, the signal processingpipeline may be implemented using other architectures than a data flowgraph with executable nodes, such as a series of purely linear(non-branching) processing sequences originating from single inputchannels and terminating at single output channels. The pipeline may usedifferent architectures to achieve all the same benefits of a data flowprocessing network, or may instead use different architectures to offera reduced or different feature set (potentially to reduce cost orsimplify the user experience). For example, a limited implementation ofthe pipeline may only offer the option of routing input channels tooutput channels, and only apply signal transformations as specified bysystem defaults and device profiles.

Example Transformations

Affine Scaling

The system may construct signals of a given range or magnitude based oninput signals of different range or magnitude via affine scaling. Thesystem may scale an input signal by first mathematically adding to itsvalue a first real number to yield an offset signal, and thenmathematically multiplying the offset signal with a second real numberto yield a scaled signal. The system may permit a user to select eitheror both of the first real number and the second real number. Themathematical operations may be executed by one or more of the system'sprocessing units. The system may use the value of the scaled signal inconstructing an output signal, or the system may store, further process,or otherwise use the scaled signal. For example, consider a userinteracting with an input device such that the user normally achieves 60degrees of angular range of motion (i.e. the angular displacement fromthe extreme of flexion, adduction, or other direction of motion, to theextreme of extension, abduction, or other opposing direction of motion),and such that that user's angular range of motion is centered at a 45degree measured angle on the input device. For this example user,scaling may be used to construct an output signal ranging from zero toone (for example to be used as a normalized output signal) using a firstreal number of negative 45 and a second real number of 60, such that byfirst adding negative 45 to the input signal and next multiplying theresulting shifted signal by 1/60, the resulting scaled signal will liein the range from 0 to 1, inclusive.

Thresholding

The system may use thresholding to construct signals taking only finitediscrete values based on continuous-valued or discrete-valued inputsignals. The system may threshold an input signal by comparing its valueagainst one or more real numbers and constructing a thresholded signaltaking one of a finite set of values according to the results of thecomparisons. The system may perform comparison operations sequentiallylow-to-high, high-to-low, or in another suitable order, or concurrently,and may execute comparison operations using one or more processing unitsor other electronic means. The system may use the value of thethresholded signal in constructing an output signal, or the system maystore, further process, or otherwise use the signal.

For example, consider a user interacting with an input device yieldingat least one continuous-valued signal such that minimum exertion by theuser yields a signal value of zero and maximum exertion by the useryields a signal value of 100. Assume this user will control a button viathis user's exertion, such that only three-quarters exertion or moretriggers the button. The system may construct a thresholded signal suchthat its value is one (logical true) when the input signal is greaterthan 75, and zero (logical false) otherwise. The system may use thisthresholded signal to construct an output signal emulating the output ofa standard switch.

In the aspect described above, thresholding could for example beimplemented as a Transformation node with a “val” input, a “threshold”input, and a “val” output, which outputs a value of “1” on the “val”output if the “val” input exceeds the “threshold” input, and a value of“0” on the “val” output otherwise.

Hysteresis

The system may introduce hysteresis during signal thresholding forpurposes such as curtailing oscillations of the resulting signal,shaping certain user experience characteristics, and other purposes. Thesystem may implement hysteresis using one or more processing units byvarying the real values used in comparison according to a previous valueof the signal being thresholded and a present value of the signal beingthresholded, such that the real value is increased when the presentvalue of the signal being thresholded is greater than the previous valueof the signal being thresholded, or such that the real value isdecreased when the present value of the signal being thresholded is lessthan the previous value of the signal being thresholded. For instance,if the previous value is less than the original real value forcomparison and the present value is greater than the original value forcomparison, the system may compare the signal being thresholded againsta value equal to the original value for comparison plus 10; whereas ifthe previous value is greater than the original real value forcomparison and the present value is less than the original value forcomparison, the system may compare the signal being thresholded againsta value equal to the original value. Alternately, if the previous valueis less than the original real value for comparison and the presentvalue is greater than the original value for comparison, the system maycompare the signal being thresholded against a value equal to theoriginal value for comparison multiplied by 1.05; whereas if theprevious value is greater than the original real value for comparisonand the present value is less than the original value for comparison,the system may compare the signal being thresholded against a valueequal to the original value minus 1. The system may use any combinationof addition and multiplication and other mathematical functions tomodify the real values for comparison. The system may alternativelyimplement hysteresis using one or more processing units or by digital oranalog circuitry or other electronic means.

Logical Functions

The system may use logical functions to create discrete-valued signalsbased on discrete-valued signals. The system may use boolean logicoperations to construct boolean signals based on boolean signals. Thesystem may use higher-order logic functions to construct signals takingone or more values from signals taking two or more values. The systemmay execute logical operations using one or more of: electronic logicgates, one or more processing units, or other digital electroniccomponents. For example, consider two buttons each producing a booleansignal taking the values 1 (logical true) when the button is pressed,and 0 (logical false) otherwise. The system may construct a booleansignal taking the values 1 (logical true) when both buttons are pressed,and 0 (logical false) otherwise, by combining the two buttons' signalsusing the logical AND operation, executed by one or more processingunits running executable code. As a separate example, consider twolever-like input devices each producing a discrete-valued signal, L1 forthe first lever, and L2 for the second lever, taking the value 0 whenpushed near to the far extreme of its range of motion, taking the value2 when pushed near to the other extreme of its range of motion, andtaking the value 1 otherwise. To measure the manual coordination of auser manipulating these levers, the system may logically combine signalsL1 and L2 to create a resulting discrete-valued signal taking the value0 when both levers are pushed near to the far extremes of theirrespective ranges of motion, 2 when both levers are pushed near to theother extremes of their respective ranges of motion, 1 when neitherlever is pushed near an extreme of its respective range of motion, and 3when each lever is in a different state from the other, by executing onone or more processing units running executable code the followinglogical function: 0 if L1==0 and L2==2; 1 if L1==1 and L2=1; 2 if L1==2and L2==2; 3 by default.

Arbitrary Functions

The system may construct signals based on other signals via functionsthat map one or more values to one or more values parametrically,discretely, piecewise, dynamically, linearly, nonlinearly, identically,exponentially, trigonometrically, or via other functions that may beimplemented or specified via software or hardware, individually or incombination. The system may permit such functions may be defined via oneor more of: standard or non-standard mathematical functions, one or moretabulations, set notation, piecewise construction, memoryless or dynamictransfer functions, or other means for defining functions and mappings,such that a resulting signal may be constructed based on a signal. Foreach value of a first signal, the system may generate a correspondingvalue of a resulting signal using one or more of: one or more digitalprocessing units, digital or analog circuitry, or other electronicmeans. The system may use executable code in conjunction with one ormore processing units to effect part or all of the generation of valuesof the resulting signal, and may make use of one or more lookup tables,standard or non-standard implementations of mathematical functions,if-else or switch-case constructs, or other coding techniques useful insignal processing. This may enable mapping functions to be designed thatcompensate for certain physical characteristics of a user, to compensatefor characteristics of an input device, to affect certain userexperience characteristics, or for other purposes. For example, considera user with muscle spasticity or tremors interacting with a lever-likeinput device that produces an input signal corresponding to the angulardisplacement of the lever, with a signal value of 0 corresponding to oneextreme of angular displacement and 10 corresponding to the otherextreme of angular displacement. The system may filter the input signalusing a 1st-order low-pass dynamic filter, represented as a transferfunction in the Laplace Domain and implemented digitally in the Z-Domainafter conversion via Z-transform, to construct a signal whereinhigh-frequency user inputs are attenuated.

Composition of Functions

The system may permit signal manipulation functions such as scaling,thresholding, linear combination, and others described in this documentto be composed such that the one or more signals resulting fromapplication of a first function are used as input signals in theapplication of a second function, and so on with two or more functions,resulting in a final resulting signal. The system may permit one or morefunctions to be composed symbolically to produce a composite function,and the system may permit a composite function to be applied to a signalto construct a resulting signal. Composition may be implementedalgorithmically in code, or by arrangement of one or more processingunits, or by arrangement of other electronic components or devices. Forexample, the system may process signals from a joystick producingseparate signals corresponding to horizontal displacements and verticaldisplacements, with values x and y, respectively, being used as an inputdevice, using executable code and one or more processing units, toemulate six separate buttons as follows: first the system may normalizethe x and y signals in the cartesian space by subtracting from each itscenter value and dividing each by its magnitude, yielding the cartesianspace values X and Y, respectively; next the system may convert X and Yto values in a polar space by setting radius r=sqrt(X{circumflex over( )}2+Y{circumflex over ( )}2), and angle theta=atan 2(X, Y); next, thesystem may threshold r using a comparison value of 0.75, to create asignal R={1 if r>0.75; 0 else}, and the system may threshold theta inincrements of 60 degrees beginning at zero, to create a signal Z={0 if0≤theta<60; 1 if 60≤theta<2*60; . . . ; 4 if 4*60≤theta<5*60; 5 else};next the system may construct six separate signals Q[i] such thatQ[i]={1 if Z=i and R=1; 0 else}, each of which may be used to emulate asingle switch.

Linear Combination of Signals

The system may permit signals to be combined via linear combination toproduce resulting signals based on the combined signals. To combine aset of first signals via linear combination, a the system may calculatea resulting signal as follows: the system enables one real number to beselected per first signal, the value of each real number selectedindependently from the others; the system may multiply the value of eachfirst signal by one such real number to produce a set of intermediatesignals; the system may set the value of the resulting signal equal tothe sum of the values of the intermediate signals (i.e. the value of thefirst intermediate signal plus the value of the second intermediatesignal, etc., plus the value of the last intermediate signal), plus thevalue of another independently selected real number. The system mayexecute linear combination using one or more of the following: digitalcircuitry, analog circuitry, one or more processing units, executablecode, or other electronic means.

Polynomial Combination of Signals

The system may combine a set of first signals via polynomial combinationto produce resulting signals based on the combined first signals. Tocombine a set of first signals via polynomial combination, the systemmay construct a resulting signal as follows: the system may multiplyeach first signal with unity or with one or more first signals(including itself) one or more times to produce a set of firstintermediate signals; the system may permit one real number to beselected per first intermediate signal, the value of each real numberselected independently from the others; the system may multiply thevalue of each first intermediate signal by one such real number toproduce a set of second intermediate signals; the system may set thevalue of the resulting signal equal to the sum of the values of thesecond intermediate signals (i.e. the value of the first of the secondintermediate signals plus the value of the second of the secondintermediate signals, etc., plus the value of the last of the secondintermediate signals), plus the value of another independently selectedreal number. For example, consider a lever-like input device producing acontinuous signal varying in response to user manipulation to yield asignal of value zero when the user does not exert and of value 100 whenthe user exerts maximum effort, and an input device measuring quality ofposture of a user and producing a continuous signal corresponding to itsmeasurement, this signal's value lying between zero and one, both inputdevices connected to the system. The posture measuring input device maybe made to inhibit the user's effectiveness with the lever-like inputdevice when the user's posture is poor by instructing the system toconstruct a signal equal to the signal of the lever-like input devicewhen the user's posture is best, decreasing in magnitude as the user'sposture becomes worse, and equal to zero when the user's posture isworst. The system may permit this to be done by allowing for the systemto be instructed to construct a resulting signal via polynomialcombination, such that the resulting signal is equal to the product ofthe lever-like input device signal and the posture measuring inputdevice signal. The system may implement polynomial combination ofsignals using one or more processing units running executable code or byone or more processing units or by other electronic means.

Mixed Combination of Signals

The system may combine signals via mixed combination to produceresulting signals based on the combined signals. To combine a set offirst signals via mixed combination, the system may construct aresulting signal as follows: the system may set a resulting signal equalto one first signal when the value of a controlling first signal isabove a set threshold, and the system may set the resulting signal equalto another first signal when the value of the controlling first signalis below the threshold. The system may repeat, cascade, or otherwisecompose such operations to yield more complex behaviors such asmultiplexing between multiple first signals according to the value ofone or more first signals. The system may execute mixed combination ofsignals using one or more processing units or by other electronic means.The system may use executable code to implement mixed combination in aflexible or predefined manner. For example, consider three button inputdevices each producing a boolean signal B[i], each button with anassociated lever input device producing a signal S[i] corresponding todeflection of the lever, all connected to the system. The system mayconstruct a resulting signal via mixed combination that takes the valueof the signal of each lever's signal when that lever's associated buttonis pressed, and value zero otherwise, by executing code on a processorunit implementing the following function: S1 if B1==1; S2 if B2==1; S3if B3==1; 0 otherwise.

Signal Limiting

The system may construct signals limited in one or more of their extremevalues based on signals via signal limiting. The system may use signallimiting to limit one or more of the maximum or minimum values of theresulting signal. The system may accomplish signal limiting by comparingthe value of a first signal against a real value, and constructing aresulting signal equal to the first signal if the first signal is eitherless the real value in the case of maximum limiting, or greater than areal value in the case of minimum limiting, and equal to the real valueotherwise. The system may compose signal limiting operations to limitone or more extremes of a signal's values. The system may executelimiting of signals using one or more processing units or by otherelectronic means. The system may use executable code to implement signallimiting in a flexible or predefined manner. For example, consider oneaxis of a joystick input device presenting a signal ranging in valuefrom 0 to 100, but being used in the central 25 degree angular arc ofits motion via affine scaling such that an intermediate signal S1 mightrange in value from −200 to +200 upon full deflections of the joystick.In order to present an output signal S0 ranging from −50 to +50 upondeflection of the joystick to the limits of its 25 degree range, thesystem may use signal limiting by executing on one or more processingunits executable code implementing the following logic: S0=S1; if S1>50,S0=50; if S1<−50, S0=−50.

Input-Triggered Macros

The system may generate, trigger, or queue macro signals embodyingpredefined sequences of values upon detection of particular values orsequences of values of one or more input signals. Macro signals mayembody constant, linear, polynomial, exponential, logarithmic,trigonometric, discrete, smooth, piecewise, or other time series ofvalues that may be useful in controlling output devices or for otheruses by or within the system. The system may generate macro signalsusing one or more processing units or by other electronic means, whichmay be complemented by the system's use of one or more of: algorithmsimplemented in executable code, one or more lookup tables, one or morememory units, or other means for generating predefined sequences ofvalues. The system may use macro signals to enact sequences ofhigh-order events in an attached output system (e.g. setting up orstarting a video game on a computer or game console). For example, thesystem may permit a button input device to be used to trigger the setupof a game on a game console output device by triggering upon the push ofthe button (a logical true signals) one or more macro signals thatemulate game controller interactions such as the press of an ‘A’ button,the leftward movement and return to neutral of a joystick horizontalaxis, the push of a ‘start’ button, and the pull of a right trigger, ofappropriate durations and sequence to direct the output device to set upand start the game.

System Configuration and UI

API and Configurability

The purpose of the system is to flexibly support a wide range of userneeds, and toward this end the system exposes configuration options thatallow users to modify its pipeline as desired for various combinationsof input devices, output devices, and signal transformations to supporta variety of use cases. These configuration capabilities are describedgenerically as an “application programming interface” (API), a set ofinstructions a user (whether a person or another system) can generate(possibly via a graphical interface) and send to the system via somemeans of communication (such as TCP packets over a network), that willcorrespondingly modify or query the system in various ways and send backrelevant information as appropriate. The API may also “push” informationto the user in response to an external data event or internal timer,without waiting for a user request. The API may also allow the user toconfigure the push notifications they would like to receive.

The essential user configuration capability is to modify of the pipelinegraph as appropriate to various uses and users. Pipeline modificationinstructions 204 may be on the level of addition, removal, and queryingof executable nodes and connections between nodes, and may be higherorder combinations of such operations for user convenience or clarity.Beyond pipeline configuration, other system configuration options may beprovided, such as, for example, network settings and logging frequency.

The system API may support a variety of methods of user configuration,such as graphical user interfaces, command line interfaces, scripts, orother applications or systems. Any interface that can produce or receiveAPI commands recognized by the system may be used as a means ofconfiguring the system.

Function Specification (Native Editor, Downloadable Plugins, ExposableAPI)

The system may permit signal mappings and signal transformationfunctions such as those described in this document or others to bespecified, loaded, tuned, defined, edited, or otherwise configured usingan interface that the system may provide, downloaded to the system froma separate source, or otherwise defined using the system or transferredto the system. The system may permit mappings and functions to be storedin one or more volatile or nonvolatile memory units or to be representedin mathematical notation, executable code, algorithm, one or more markuplanguages, or other method of storing mappings or functions. The systemmay permit mappings and functions to be created by recombining orediting previously defined mappings or functions. The system may permitmappings and functions to be specified by inputting or transferringmathematical, logical, or other formulas graphically, by command line,or by other means of interface. The system may permit mapping andfunctions to be specified by inputting or transferring one or morelookup tables in tabular form, as a markup document, or other format forspecifying lookup tables. The system may permit functions to bespecified by creating or transferring parametric or discrete-valuedinput-output curves, such as linear, polynomial, bezier, spline, andother types of curve. The system may supply a graphical user interfacethat may be used in constructing such curves. The system may use linearor nonlinear interpolation methods to interpolate between specifiedpoints in the definition of a function. The system may permit mappingsand functions to be specified such that one or more of their definingvalues may be chosen or altered by a user, by the system, or by othercommand, such that the mapping or function may be executed to reflectthe defining values. The system may supply one or more linear sliders,rotational knobs, or other variable interface component, or virtualgraphical depictions thereof, that may be used in adjusting the definingvalues. The system may expose one or more application programminginterfaces that may enable the use of various specifications,implementations, and representations of mappings or functions, parts ofmappings or functions, or mathematical or other operations. The systemmay embody mapping or function specification or tuning modes enablinginteractive definition of mappings, functions or defining values. Thesystem may measure characteristics of input device signals receivedduring specification or tuning to select defining values, such asmeasuring a user's range of motion, strength, speed, dexterity, or othercharacteristics to select defining values for scaling, thresholding, orother functions that may influence the user's experience. The system mayenable specification, loading, tuning, defining, editing, and otherconfiguration functionalities using one or more processing units or byother electronic means, which may be complemented by the use of one ormore of: algorithms implemented in executable code, one or moregraphical displays, one or more graphics specifications or predefinedgraphical views, one or more memory units, or other means for enablingthe functionalities, which may be part of the system. The system may useone or more wired or wireless connections to enable thesefunctionalities to be executed remotely.

The system may permit macro signals such as those described in thisdocument or others to be specified, loaded, tuned, defined, edited, orotherwise configured using an interface that may be provided by thesystem, downloaded to the system from a separate source, or otherwisedefined using the system or transferred to the system. The system maystore macro signals in one or more volatile or nonvolatile memory units,and may permit their representation in mathematical notation, executablecode, algorithm, one or more markup languages, or other method ofstoring macro signals. The system may permit macro signals to be createdby recombining or editing previously defined macro signals. The systemmay permit macro signals to be specified by inputting or transferringmathematical, logical, or other formulas or time series graphically, bycommand line, or by other means of interface. The system may permitmacro signals to be specified by inputting or transferring one or morelookup tables in tabular form, as a markup document, or other format forspecifying lookup tables. The system may permit macro signals to bespecified by creating or transferring parametric or discrete-valuedinput-output curves, such as linear, polynomial, bezier, spline, andother types of curve. The system may embody a graphical user interfacethat may be used to construct such curves. The system may use linear ornonlinear interpolation methods to interpolate between specified pointsin the definition of a macro signal. The system may permit macro signalsto be specified such that one or more of their defining values may bechosen or altered by a user, by the system, or by other command, suchthat the macro signal may be generated in a way reflective of thedefining values. The system may provide one or more linear sliders,rotational knobs, or other variable interface component, or virtualgraphical depictions thereof, which may be used in adjusting thedefining values. The system may expose one or more applicationprogramming interfaces allowing for the use of various specifications,implementations, and representations of macro signals, parts of macrosignals, or mathematical or other operations. The system may embodymacro signal specification or tuning modes enabling interactivedefinition of macro signals or defining values. The system may measurecharacteristics of input device signals received during specification ortuning to select defining values, such as by recording a sequence ofuser interactions to establish a macro signal for performing the taskperformed by the user during the recording. The system may enablespecification, loading, tuning, defining, editing, and otherconfiguration functionalities using one or more processing units or byother electronic means, which may be complemented by the system's use ofone or more of: algorithms implemented in executable code, one or moregraphical displays, one or more graphics specifications or predefinedgraphical views, one or more memory units, or other means for enablingthe functionalities, which the system may embody. The system may use oneor more wired or wireless connections to enable these functionalities tobe executed remotely. For example, to create a macro signal that may beused to set up a game on a game console output device, the system mayprompt a user via one or more graphical displays to manipulate inputdevices in a way and sequence that, according to the current set ofmappings and functions, result in the desired setup in an output device;the system may record the resulting output signals for later use as amacro signal that may be later used with the same mappings and functionsto set up a game. The system may begin recording upon detection of afirst manipulation by the user (i.e. when one or more signals from oneor more input devices take values other than their rest values), and thesystem may end recording after a preset amount of time lapses followinga final manipulation by the user. The system may initiate or terminaterecording upon specific action by a user, such as the push of a buttonor a virtual button. The system may dilate, contract, or normalizerecorded sequences in time via signal processing methods or othermethods useful in altering signals.

Visualization and Feedback

The system may provide one or more forms of feedback to the userreflecting the state or configuration of the system or interactions ofthe user with input devices or with the system. The system may providefeedback via one or more of: a visual display, audible, vibrotactile,haptic, or other feedback, which the system may embody, or which thesystem may connect to remotely via a wired or wireless connection, orvia a communications network. The system may, for example, providefeedback to a user reflecting the quality, intensity, or other parameterof the user's interaction with input devices. For example, the systemmay present an LED, or other light or virtual mark, the color, state,brightness, or other characteristics of which, as the system detectsvarious levels of the parameters, the system may vary according to thedetected levels. The system may, for example, provide feedbackreflecting the resultant signals after mappings, transformations, orfunctions may be applied. The system may provide feedback to a user thatthe user may use while the user grows accustomed to a particularconfiguration or mapping. For example, the system may display an avataror other representation of an arm or another body part, or another otherfigure, the form, color, or other characteristics of which figure thesystem may alter in response to user interaction with the input devices.The system may provide feedback to a user while the user is using thesystem to control one or more output devices by interacting with one ormore input devices, which may provide the user a more direct responsethan is decipherable via the response of the output systems.

User Profiles

The system may permit the creation and management of one or more userprofiles that may aggregate data or metadata associated with a givenuser, and may include one or more mappings from input devices to outputdevices, transformations, functions, or mappings to be applied tosignals, identifications of one or more signals to be recorded, stored,or otherwise processed or used, or other data or metadata relevant tothe user. The system may permit a user to select a user profile, and mayupon selection by a user configure the system according to thespecifications of the selected profile. The system may store profilesusing one or more memory units. The system may represent the profiles inone or more formats, such as markup documents, JavaScript ObjectNotation (JSON), EXtensible Markup Language (XML), or other formats thatmay be used to represent profiles. The system may permit profiles to becreated, modified, recombined, and otherwise manipulated via the system,possibly using one or more graphical interfaces which the system mayembody. The system may permit profiles to be transferred to or from thesystem via wireless or wired connection. For example, the system maystore as a profile information from one user session that may be used torapidly configure the system for a future user session.

System Modification and Updates

It may be desirable from time to time to modify or replace existingexecutable code on the system or to transfer new executable code to thesystem. Such executable code may fall into several categories, which mayinclude system firmware and software, communication protocol drivers,signal processing functions and other functions for modifying input orintermediate signals, output system start-up macros, input-triggeredmacros, or other system macros. The system may store such executablecode in one or more persistent or non-persistent memory stores, whichmay consist of program memory or data or other memory. Such memory storemay be previously unused or may be previously used by other driver code,system code, macro code, a combination thereof, or any other data. Thesystem may execute instructions directly using one or more processingunits, or may load instructions during runtime and execute them asnative instructions using one or more processing units, or may loadinstructions during runtime and interpret them as non-nativeinstructions. The system may permit drivers to be transferred to or fromthe system using a wired or wireless connection, which transfer may beinitiated by a user via a graphical interface, a command line interface,or another user interface, or may be initiated by an automated process.

The system may provide a means for installing drivers that implementlower order communication protocols (e.g., USB), or that implementhigher order communication protocols (e.g., MMC/SD) that are based uponlower order communication protocols, which may enable the system tosupport communications protocols that it may not have supported whenmanufactured and initially programmed. Drivers may include executablecode that may be transferred to the system and executed by such meansdescribed in this document. As an example, it may be that an aspect ofthe system does not natively support communication with an inertialmeasurement unit (IMU) that sends and receives data using a customhigher-level protocol that is transmitted via the lower-level SPIprotocol. If the system has or is supplied with a hardware SPI portcapable of maintaining an SPI connection with the IMU, instead ofrequiring an external adapter to interconvert the signals from and tothe IMU so that the system might communicate with it as an input device,a driver may be uploaded to the system to interconvert the signals fromand to the IMU. As another example, it may be that an aspect of thesystem does not natively support communication with a game system thatsends and receives data using a custom higher-level protocol that istransmitted via a custom lower-level protocol. If the system has or issupplied with a hardware port capable of maintaining a connection withthe game system, instead of requiring an external adapter tointerconvert the signals from and to the game system so that the systemmight communicate with it as an output device, a driver may be uploadedto the system to interconvert the signals from and to the game system.

Extensions for Training, Rehabilitation, and Collaborative Use

Overview

This section describes system functionalities for facilitating training,rehabilitation and collaborative control of systems. Generally speaking(as elaborated further in the following sections), the system may beused to implement various mappings from various input devices to outputdevices in such a manner as to encourage a user to learn or improve oneor more movements or skills of interest. These devices and mappings maybe selected and mapped via the system (using features described in thisdocument) for example by a skilled heavy equipment operator, aircraftoperator, physical therapist, occupational therapist, caretaker, orother person or entity, so as to assist, encourage, prompt, or otherwisefacilitate a patient, trainee, or other user's learning or improving ofmovements or other modes of input device interaction that may be relatedto the function of a connected output device being controlled via thesystem. The focus may for example be to train a user to skillfully ormore skillfully operate an output system, skill being measured byefficacy, efficiency, safety, or any other desired metric or combinationof metrics.

For example, consider a new employee learning how to operate a gantrycrane. A skilled equipment operator may for example use the system tomap redundant control panels to operate the gantry crane, giving the newemployee primary control of the crane, and giving herself secondaryoverride operation of the crane. In this way the skilled equipmentoperator may guide the new employee in operating the gantry crane andmay for example better ensure that accidents are prevented during theearly phases of training.

The focus may also be to help the user to recover or learn capabilitiesbeyond skillful control of a particular output system (e.g., physicalcapabilities such as range of motion, strength, speed, or dexterity;mental capabilities such as quickness of response or improved short-termmemory retention; other capabilities including but not limited toauditory and visual capabilities). For physical rehabilitation, thismethod has been previously employed with specific input devicesconnected to a specific output device—for example, the ArmeoSpring byHocoma has an arm exoskeleton connected to a virtual display encouraginga user to perform various therapy-relevant movements—but the method hasnot been previously employed with an intermediary system that allows oneor more input devices to be connected to one or more output systems inmultiple configurations as desired (as embodied by the system disclosedin this document), which allows more flexibility in matchingrehabilitation or training-relevant types of user interaction toappropriate or engaging output devices. In the course of rehabilitationor training the user may benefit from assistance in either or both ofinteracting with the input device and controlling the output device.There are a number of methods of assistance (as elaborated further inthe following sections) that may extend the rehabilitation methoddescribed. These may include using supplemental input devices notintended primarily for rehabilitation or training purposes, supplementalinput devices for simultaneous use by other users (the modes of use ofwhich may include complementary, overriding, additive, or somecombination thereof), and force-assistive input devices for directassistance. In the following sections examples are centered mostlyaround a particular type of rehab exercise (arm range-of-motiontraining), a single virtual interactive display unit (the NintendoGameCube game console) and a single application being executed on thevirtual interactive display unit (Super Smash Bros.) to illustrate thedifference between various assistance modes; however, each mode is notlimited to any particular virtual interactive display unit orapplication executing on such display unit, and can apply to variouscombinations of rehab exercises or other movements, input devicescapturing such movements, output devices, and applications executing onsuch output devices, provided the input and output devices may be usedwith the system described in this document. As a baseline example,consider a stroke patient who is hemiparetic (left-impaired), withlimited use of the left arm but proficient use of the right arm. Atherapist may for example use the system to map a passive device forrange-of-motion exercises (such as the ArmeoSpring arm exoskeleton) withthe patient's left arm to the left and right arrow keys of a computerkeyboard, and may for example decide to load the game Breakout (a gamewith a sliding paddle used to keep a ball from falling off of thescreen) on a computer. For example, it may be that the therapist sets athreshold such that when the angle of the elbow joint of the exoskeletonis less than 75°, the right arrow key is triggered, and when the angleof the elbow joint of the exoskeleton is more than 105°, the left arrowkey is triggered. Using the system, the therapist may also for examplereplace the ArmeoSpring with another input device relevant to therapy(such as a knee range-of-motion exerciser), which decision may forexample be made according to a therapy regimen developed by thetherapist, and may also for example replace the game Breakout withanother game on another output device supported by the system, whichdecision may for example be made in order to maintain patient engagementafter the patient tires of a particular game.

Patient Supplemental Controls

A desired application to be controlled (such as a video game) running ona connected output device may require more input channels for controlthan can reasonably be captured via rehabilitation-specific ortraining-specific movements or other forms of interaction. The systemmay allow a patient or other user using one or more input devices orchannels intended primarily for rehabilitation or other trainingpurposes to also use one or more other input devices or channels notintended primarily for rehabilitation or other training purposes,possibly for the purpose of enhancing the user's ability to utilize theoutput device being controlled via the system. For example, consider astroke patient who is hemiparetic (left-impaired), with limited use ofthe left arm but proficient use of the right arm. A therapist may forexample use the system to map a passive device for range-of-motionexercises (such as the ArmeoSpring) with the patient's left arm to thejoystick input of a Nintendo GameCube game controller, and may forexample decide to load the game Super Smash Bros. (a platform fightinggame) on a Nintendo GameCube game console. For example, it may be thatthe therapist sets a mapping such that angle range of the elbow joint ofthe exoskeleton from 75° to 105° maps linearly to the full dynamic rangeof the game controller joystick (from right to left). With this mappingalone, the patient may be able to move a character in the game butcannot perform other in-game functions such as “attacks” or game consolefunctions such as menu navigation. In order to facilitate a fullergameplay experience, the therapist may give the patient a supplementalgame controller connected via the system to the output device that thepatient may use with the right hand. This may for example enable thepatient to perform attacks and menu navigation while performingrehab-specific movements to move a player in the game.

Therapist/Trainer/Caretaker Supplemental Controls

A desired application to be controlled (such as a video game) running ona connected output device may require more input channels for controlthan a patient can manage, for reasons that may include the complexityof their rehabilitation or training setup, insufficient physicalability, other limitations of ability, or some combination thereof. Thesystem may allow a therapist, trainer, caretaker, family member, friend,or other person to use input devices alongside the input devices used bya patient or other user who is using the system for rehabilitation orother training purposes. For example, consider a stroke patient who ishemiparetic (left-impaired), with limited use of both arms. A therapistmay for example use the system to map a passive device forrange-of-motion exercises (such as the ArmeoSpring) with the patient'sleft arm to the joystick input of a Nintendo GameCube game controller,and may decide to load the game Super Smash Bros. (a platform fightinggame) on a Nintendo GameCube game console. For example, it may be thatthe therapist sets a mapping such that angle range of the elbow joint ofthe exoskeleton from 75° to 105° maps linearly to the full dynamic rangeof the game controller joystick (from right to left). With this mappingalone, the patient may be able to move a character in the game but willbe unable to perform other in-game functions such as “attacks” or gameconsole functions such as menu navigation. In order to facilitate afuller gameplay experience, the therapist may use a supplemental gamecontroller connected via the system to the game console to play the gamecollaboratively with the patient. This may for example allow the patientto perform more successfully in the game, thus better engaging andmotivating the patient.

Therapist/Trainer/Caretaker Override Controls

A desired application to be controlled (such as a video game) running ona connected output device may require more input channels for controlthan a patient can manage, for reasons that may include the complexityof their rehabilitation or training setup, insufficient physicalability, other limitations of ability, or some combination thereof. Thesystem may allow a therapist, trainer, caretaker, family member, friend,or other person to use input devices alongside the input devices used bya patient or other user who is using the system for rehabilitation orother training purposes.

For Example, Consider a Stroke Patient Who is Hemiparetic(Left-Impaired), with Severely Limited use of their left arm butmoderate use of their right arm. A therapist may decide to use thesystem to map a passive device for range-of-motion exercises (such asthe ArmeoSpring) for the patient's left arm to the joystick input of agame system controller, and may decide to load the game Super SmashBros. (a platform fighting game) on a game system. For example, it maybe that the therapist sets a mapping such that angle range of the elbowjoint of the exoskeleton from 75° to 105° maps linearly to the fulldynamic range of the game controller joystick (from right to left). Thetherapist may also decide to give the patient a supplemental gamecontroller that the patient can use with their right hand, to enable thepatient to perform attacks and menu navigation. The therapist may alsoconfigure and assign macros to buttons or combinations thereof on thepatient's right-hand controller in order to enable more complex attacksthan the patient is capable of triggering on their own. Despite theseaccommodations, the patient may still become frustrated because of theirlimited ability or limited experience with the game. The therapist maydecide to configure a supplemental joystick controller via the system asan override for the patient joystick output. The therapist may thenoverride the patient's joystick control with the therapist's control bymoving this supplemental joystick past a specified threshold, and maythus assist the patient in performing more skillfully in the game whichmay better engage and motivate the patient.

Therapist/Trainer/Caretaker Additive Controls

The system may permit configuration such that input provided by atherapist or other person via one or more input devices or anothersystem, or transformations thereof, may be added to (summed with orlogically combined with, as appropriate) that of a patient or other userbefore, within, or after mapping.

For example, consider a stroke patient who is hemiparetic(left-impaired), with severely limited use of their left arm butmoderate use of their right arm. A therapist may decide to use thesystem to map a passive device for range-of-motion exercises (such asthe ArmeoSpring) for the patient's left arm to the joystick input of agame system controller, and may decide to load the game Super SmashBros. (a platform fighting game) on a game system. For example, it maybe that the therapist sets a mapping such that angle range of the elbowjoint of the exoskeleton from 75° to 105° maps linearly to the fulldynamic range of the game controller joystick (from right to left). Thetherapist may also decide to give the patient a supplemental gamecontroller that they can use with their right hand, to enable thepatient to perform attacks and menu navigation. The therapist may alsoconfigure and assign macros to buttons or combinations on the patientsright-hand controller in order to enable more complex attacks than thepatient is capable of triggering on their own. Despite theseaccommodations, the patient may still become frustrated because of theirlimited ability or limited experience with the game. The system maypermit the therapist to configure a supplemental joystick controller asan additive supplement to the patient joystick output. The therapist maythen adjust the patient's joystick control when desired by moving thethe supplemental controller from its rest position, and may thus assistthe patient in performing more skillfully in the game which may betterengage and motivate the patient.

Haptic or Other Input-Derived Assistance

The system may permit signals from one or more first input devices to bemapped and sent to one or more second input devices that support receiptof data. This may facilitate modes of operation wherein a therapist orother first user may guide or otherwise shape the interactions of apatient or other second user. This may also facilitate modes ofoperation wherein one input devices measuring user characteristics maybe used to guide or otherwise shape the interactions of that user. Thesystem may represent or handle such signals as it handles other signals,or may distinguish them from others or handle them differently.

For example, consider a stroke patient who is hemiparetic(left-impaired), with severely limited use of their left arm andmoderate use of their right arm. A therapist may decide to use thesystem to map an active force-assistive rehab device for range-of-motionexercises (such as the ArmeoPower) for the patient's left arm to thejoystick input of a game system controller, and may decide to load thegame Super Smash Bros. (a platform fighting game) on a game system. Forexample, it may be that the therapist sets a mapping such that anglerange of the elbow joint of the exoskeleton from 75° to 105° mapslinearly to the full dynamic range of the game controller joystick (fromright to left). The therapist may also decide to give the patient asupplemental game controller that they can use with their right hand, toenable the patient to perform attacks and menu navigation. The therapistmay also configure and assign macros to buttons or combinations on thepatient's right-hand controller in order to enable more complex attacksthan the patient is capable of triggering on their own. Despite theseaccommodations, the patient may be unable yet to engage in meaningfultherapy exercise or gameplay on their own given their limited armfunction. The therapist may use the system to configure a supplementaljoystick controller (either haptic or passive) as a master controllerfor the force-assist rehab device, as well as replacing the patient'sjoystick output with the output of the supplemental joystick controller.In this configuration, the therapist is the one actually controlling thejoystick input to the game, while the patient feels the movements usedby the therapist in playing the game via the force display of theforce-assistive rehab device as controlled by the therapist's joystick.This configuration may be used as an aid for teaching the patientdesired movements without hands-on physical assistance when the patientdoes not have enough function to meaningfully engage in rehab-specificexercises or gameplay unassisted.

Alternately, the therapist may use the system to configure asupplemental joystick controller (either haptic or passive) as a mastercontroller for the force-assist rehab device, while letting thepatient's joystick output continue to be the joystick input to the game.In this configuration, the patient is the one actually controlling thejoystick input to the game, while feeling the movements used in thetherapist's simultaneous gameplay via the force display of theforce-assistive rehab device as controlled by the therapist's joystick.This configuration may be used as a way to assist the patient withdesired movements without hands-on physical assistance when the patientdoes not have enough function to skillfully engage in rehab-specificexercises or gameplay unassisted.

As a separate example, consider a stroke patient who is hemiparetic(left-impaired), with severely limited use of their left arm andmoderate use of their right arm. A therapist may decide to use thesystem to map an active force-assistive rehab device for range-of-motionexercises (such as the ArmeoPower) for the patient's left arm to thejoystick input of a game system controller, and may decide to load thegame Super Smash Bros. (a platform fighting game) on a game system. Forexample, it may be that the therapist sets a mapping such that anglerange of the elbow joint of the exoskeleton from 75° to 105° mapslinearly to the full dynamic range of the game controller joystick (fromright to left). The therapist may also decide to give the patient asupplemental game controller that they can use with their right hand, toenable the patient to perform attacks and menu navigation. The therapistmay also configure and assign macros to buttons or combinations on thepatient's right-hand controller in order to enable more complex attacksthan the patient is capable of triggering on their own. Despite theseaccommodations, the patient may be unable yet to engage in meaningfultherapy exercise or gameplay on their own given their limited armfunction. The system may enable the therapist to use EMG electrodesaffixed to the patient to record the patient's bicep, tricep, or othermuscle activation, and configure the system to use the EMG input as a aninput to the system, which may be sent to the force-assistive rehabdevice or one or more other input devices or one or more output devices.In this configuration, the effort of the patient to move their arm maybe amplified with assistance from the force-assist rehab device based onthe intended movement as inferred from processed EMG signals from thepatient's arm, which may enable the patient to practice greater range ofmotion and perform better in the game than they could withoutassistance.

As a separate example, consider a stroke patient who is hemiparetic(left-impaired), with severely limited use of their left arm andmoderate use of their right arm. A therapist may decide to use thesystem to map an active force-assistive rehab device for range-of-motionexercises (such as the ArmeoPower) for the patient's left arm to thejoystick input of a game system controller, and may decide to load thegame Super Smash Bros. (a platform fighting game) on a game system. Forexample, it may be that the therapist sets a mapping such that anglerange of the elbow joint of the exoskeleton from 75° to 105° mapslinearly to the full dynamic range of the game controller joystick (fromright to left). The therapist may also decide to give the patient asupplemental game controller that they can use with their right hand, toenable the patient to perform attacks and menu navigation. The therapistmay also configure and assign macros to buttons or combinations on thepatient's right-hand controller in order to enable more complex attacksthan the patient is capable of triggering on their own. Despite theseaccommodations, the patient may be unable yet to engage in meaningfultherapy exercise or gameplay on their own given their limited armfunction. The system may permit the therapist to configure the system touse motion analysis or force sensing of the patient's arm movements toregulate the assistance provided by the force-assist input device. Inthis configuration, the effort of the patient to move their arm may beamplified with assistance from the force-assist rehab device based onthe patient's intended movement as inferred from motion analysis orforce sensing, which may enable the patient to practice greater range ofmotion and perform better in the game than they could withoutassistance, with less cost and setup burden than that of using EMGelectrodes.

Dynamic Adjustment (Progression/Regression) of Mapping/Assistance

The system may permit a therapist or other person or system to adjustdefining values affecting mapping, configuration, or transformations viaa graphical interface that the system may embody or other means.

For example, consider a stroke patient who is hemiparetic(left-impaired), with limited use of their left arm and moderate use oftheir right arm. A therapist may decide to use the system to work onpatient range-of-motion, mapping a force-assistive rehab device (such asthe ArmeoSpring or ArmeoPower) to the joystick input of a game systemcontroller, and may decide to assign games such as Super Smash Bros. (aplatform fighting game) on a game system for the patient to play overthe course of their rehab treatment. For example, it may be that thetherapist sets a mapping such that angle range of the elbow joint of theexoskeleton from 75° to 105° maps linearly to the full dynamic range ofthe game controller joystick (from right to left). Over the course oftreatment, the therapist may manually adjust a scaling parameter(through the router mapping interface) that determines how the workingrange-of-motion of the patient should map to the dynamic range of thejoystick, and may increase the scaling as the patient improves toencourage the patient to increase their working range-of-motion. Thisadjustment may occur between sessions after an informal or formalpatient assessment at the discretion of the therapist.

The therapist may also manually adjust the scaling parameter inreal-time as the patient plays the game, as desired, for example providea short period of extra challenge or to accommodate fatigue bydecreasing challenge.

Automatic Progression

The system may automatically tune or adjust parameters (such asscalings, thresholds, and other defining values) affecting mappings,configurations, or transformations of signals in response to usereffort. The system may set or alter such parameters using open-loop orclosed-loop control. The system may set parameters to achieve goals thatmay be set by a therapist or other user or system (patient achievementas a function of time, effort, number of repetitions, distance ofmotion, or other measure of duration) by adjusting parameters toaccomplish the goals at their appointed times. The system may setparameter values linearly over time; it may incorporate a cubic spline,quintic spline, or other spline; it may allow for plateaus. For example,consider a stroke patient who is hemiparetic (left-impaired), withlimited use of their left arm and moderate use of their right arm. Atherapist may decide to use the system to work on patientrange-of-motion, mapping a force-assistive rehab device (such as theArmeoSpring or ArmeoPower) to the joystick input of a game systemcontroller, and may decide to assign games such as Super Smash Bros. (aplatform fighting game) on a game system for the patient to play overthe course of their rehab treatment. For example, it may be that thetherapist sets a mapping such that angle range of the elbow joint of theexoskeleton from 75° to 105° maps linearly to the full dynamic range ofthe game controller joystick (from right to left). Over the course oftreatment, the therapist may set start and end points over a fixedperiod of time for automated adjustment of a scaling parameter (throughthe router mapping interface) that determines how the workingrange-of-motion the patient should map to the dynamic range of thejoystick. For example, if the therapist determines that the patient hasa comfortable elbow range of motion of 45 degrees, the therapist maydecide to set a start point of 45 degrees and an endpoint of 60 degreesover the course of two weeks of therapy. The system may allow thetherapist to select from multiple options for how the system shouldprogress the range-of-motion scaling parameter, including but notlimited to linear progression, spline-based progression, and piecewiseprogressions, which may or may not adapt dynamically to the patient toaccount for plateaus, fatigue, or other factors disruptive to progressor motivation.

The system may set parameters automatically to regulate the values ofone or more measures of user effort, for example range-of-motion aspercent of full-scale, optionally taking into account present mapping orconfiguration settings, by treating such measures as setpoints incontrol schemes or other methods of regulation. The system may permitother sensors, such as heart rate or skin conductance, or processedsignals, or other indicators of user stress, impatience, frustration,fatigue, decreased performance, or other characteristics, to beconfigured to automatically adjust parameters. The system mayautomatically trigger a plateau period, decrease difficulty for theuser, or otherwise adjust configuration upon detecting certain values,sustained values, peak values, value sequences, or other features ormeasures in signals reflecting the characteristics. For example, thesystem may adjust one or more parameters using one or more open-loop orclosed-loop control schemes (e.g. proportional integral derivative (PID)control, linear quadratic regulator (LQR) control, or other controlschemes), using a quantity such as the ratio of (a) a measure of theeffort, intensity, quality, or other characteristic of patientinteraction to (b) a measure of total possible range given presentmapping or configuration (e.g. the range of input that would yieldfull-scale outputs given current mapping or configuration), or otherquantity reflecting patient interaction, as the setpoint of the controlschemes. For example, consider a stroke patient who is hemiparetic(left-impaired), with limited use of their left arm and moderate use oftheir right arm. A therapist may decide to use the system to work onpatient range-of-motion, mapping a force-assistive rehab device (such asthe ArmeoSpring or ArmeoPower) to the joystick input of a game systemcontroller, and may decide to assign games such as Super Smash Bros. (aplatform fighting game) on a game system for the patient to play overthe course of their rehab treatment. For example, it may be that thetherapist sets a mapping such that angle range of the elbow joint of theexoskeleton from 75° to 105° maps linearly to the full dynamic range ofthe game controller joystick (from right to left). The system may permitthe therapist or other person or system to configure the system forfully automated adjustment of a scaling parameter affecting the mappingof patient working range-of-motion to dynamic range of the joystick. Thesystem may permit the therapist to select from multiple options for howthe system should progress the range-of-motion scaling parameter, whichmay include linear progression, spline-based progression, piecewiseprogressions, or other progressions, which may or may not adaptdynamically to the patient to account for plateaus, fatigue, or otherfactors disruptive to progress or motivation.

Extensions for Data Collection, Analysis and Display

Data Capture—“Metadata” (User Info, Length of Session, Equipment) &Usage Data

The system may capture, store, process, transmit, or display metadataabout usage or configuration of the system. The metadata may includefrequency of use, duration of use, identifications of one or more usersor other people or systems configuring the system, identifications ofone or more users interacting with input devices, information aboutinput devices connected to the system, information about output devicesconnected to the system, information about configuration of the system(which may include information about mappings from input devices tooutput devices), annotations that may be entered or supplied by a useror system, and other information reflecting usage of the system,configuration of the system, or interaction with the system. The systemmay capture metadata continuously while it is powered, or selectivelyduring certain periods, which the system may permit to be specified byone or more users or other systems, by logging to memory certaininternal parameters which may be present as a result of other operationof the system or which may be constructed by the system for the purposeof capturing metadata or for other purposes. The system may capturevarious portions of metadata on various schedules. The system may storemetadata using one or more memory units. The system may process metadatato provide meaningful summaries thereof or for other purposes using oneor more processing units or other electronic means. For example, thesystem may compute averages (mean, median, or mode, as may be useful) ofvarious portions of metadata. The system may also, for example, computesums of various portions of metadata. The system may also, for example,construct histograms reflecting various portions of metadata that may becaptured in discrete-valued form. The system may transmit metadata ordata produced by processing metadata via one or more wired or wirelessconnections. The system may encode metadata or processed data in one ormore formats, such as markup documents, JavaScript Object Notation(JSON), EXtensible Markup Language (XML), or other formats that may beused to store, process, display, or transmit data. The system maydisplay metadata or processed data via one or more static or interactivegraphical displays the system may embody. For example, consider atherapist using the system to provide a paid service of enabling a userto interact with one or more output devices via one or more inputdevices; the system may display to the therapist metadata such asduration of use by the user, identification of the therapist,identification of the user, types and identifications of connected inputdevices, types and identifications of output devices, information aboutcurrent mappings from input devices to output devices, and othermetadata or processed data.

Data Capture Example—Patient Usage Measurements

The system may capture, store, process, transmit, or display data aboutone or more users' usage of the system or interaction with inputdevices. The data may include intensity of use, range of motion, one ormore forces exerted, speeds of exertion, dexterity, or othercharacteristics of user interaction, annotations that may be entered orsupplied by a user or system, and other information reflecting usage ofthe system or interaction with input devices. The system may capturedata continuously while it is powered, for the duration of one or moreuser sessions, or selectively during certain periods, which the systemmay permit to be specified by one or more users or other systems, bylogging to memory certain internal parameters which may be present as aresult of other operation of the system or which may be constructed bythe system for the purpose of capturing data or for other purposes. Thesystem may capture various portions of data on various schedules. Thesystem may store data using one or more memory units. The system mayprocess data to provide meaningful summaries thereof or for otherpurposes using one or more processing units or other electronic means.For example, the system may compute averages (mean, median, mode, orweighted averages, as may be useful) of various portions of data. Thesystem may also, for example, compute sums of various portions of data.The system may also, for example, apply any transformation, function, ormapping described in this document or other transformations, functions,or mappings to various portions of data. The system may also, forexample, construct histograms reflecting various portions of data thatmay be captured in discrete-valued form. The system may transmit data ordata produced by processing data via one or more wired or wirelessconnections. The system may encode data or processed data in one or moreformats, such as markup documents, JavaScript Object Notation (JSON),EXtensible Markup Language (XML), or other formats that may be used tostore, process, display, or transmit data. The system may display dataor processed data via one or more static or interactive graphicaldisplays the system may embody. For example, consider a therapist usingthe system to provide a paid service of enabling a user to interact withone or more output devices via one or more input devices; the system maydisplay to the therapist data such as duration of use by the user,intensity of use, range of motion, one or more forces exerted, speeds ofexertion, dexterity, or other characteristics of user interaction,annotations that may be entered or supplied by a user or system, andother information reflecting usage of the system or interaction withinput devices, and other usage data or processed data. The system mayhandle usage data and metadata separately or together, and may display,process, or otherwise manipulate, present, or transmit them together orseparately.

Data Capture Example—Single or Periodic Patient Assessments

The system may permit single or periodic assessments of thecharacteristics of a user interacting with one or more input devicesconnected to the system. The system may instruct or otherwise prompt auser to manipulate or otherwise interact with one or more input devices,and may capture, store, process, transmit, or display signals or one ormore parts of signals captured that may reflect one or more of theuser's interactions. The system may prompt a user to interact with theinput devices in certain ways (i.e. along certain axes, using certaindegrees of freedom, etc.) in a particular sequence, and may permit auser or other person or system to specify the sequence, and may executesteps in the sequence for particular durations, which the system maypermit a user or other person or system to specify, or automaticallyupon detection of certain characteristics of a user's interaction withone or more input devices, using one or more processing units, or otherelectronics means for sequencing, timing, detecting, and capturingsignals and prompts. For example, the system may measure user efforts interms of peaks, averages, median or other functions of any measured datapoint, which the system may permit one or more users or other persons orsystems to select, define, or specify. The recording may be done locallyon the mapping device or connected computing device, or remotely on acomputing device that is connected by a communication network. Forexample, a therapist using the system to enable a user to interact withone or more output devices using one or more input devices may wish tomeasure the development of the user's capabilities over time; the systemmay permit the therapist to periodically assess characteristics of theuser's interactions, which may be used for evaluation of the user'sprogress, and the system may also process or otherwise manipulate thedata and may display the data in graphical, tabular, or other form, orotherwise convey the data to the therapist.

Extensions for Teleoperation/Tele-Administration

The system may permit each system functionality, such as administrationof the system, system configuration, mapping, setup, recording, display,reporting, usage summary, dynamic modes, assistive modes, and othersystem functionalities, via interfaces and functionalities the systemmay provide or via remote signalling, such as via one or morecommunications networks. The system may expose one or more applicationprogramming interfaces (APIs) that may permit one or more users or otherpersons or system to interface with the system remotely. For example,the system may enable a therapist to connect to the system via acommunication network, and may permit the therapist to collect, view,download, upload, edit, configure, or otherwise interact with data ormetadata or configurations, mappings, functions, transforms, or otherstates, parameters, or aspects of the system.

The following describes a particular aspect of the invention.

Embodiment Overview

As shown in FIG. 8 , an example aspect of a “standalone system” isdescribed here, in which the system receives and interconverts data 104105 109, feeds it 801 into a signal processing pipeline 203, andinterconverts the resulting transformed data 109 105 104 for output toan output system 107 108. The pipeline can be configured using agraphical user interface (GUI) 703, which configures the system using anAPI 803 provided by the main application 801 described further below.

User Interface Devices and Output Systems Supported byHardware+Firmware/Drivers

In the example aspect, the system includes low-level hardware 104 andsoftware 105 to connect and communicate with user interface devices andoutput systems. Specific compatibility may be configured to suitparticular applications, and standard USB and Bluetooth implementations,as well as other protocols such as switch closure or customimplementations, may be used. The aspect described here has four USBports and supporting electronics 104 and software 105, allowing thesystem to support USB HID devices as user interface devices 103 (USB HIDdevices include mice, keyboards, joysticks, foot pedals, head switches,a variety of assistive devices, and many other specialty devices). Ifdesired, an off-the-shelf USB hub can be used to support more than fourdevices at once (or devices with large power requirements). The aspectdescribed here has a set of GPIO digital inputs, six of which areconnected to physical 3.5 mm jacks to support the use of 3.5 mm switchinputs (such as “buddy buttons” or certain sip-and-puff input devices)as another device type. The aspect described here uses a third-partyadapter to support USB bluetooth connectivity; however, in anotheraspect bluetooth may be built into the system. To support control of agame console 108, the system can be connected to an adapter board 107that takes serial input over USB and relays it to the game console,either by electronically activating the inputs on the circuit board of agame controller for that console, or by emulating the console controllerprotocol directly. Alternately, to support control of a PC 108, thesystem can be connected to an adapter board 107 that takes USB serialinput over USB and relays it to the computer as USB HID device.Similarly, it will be understood that in various configurations, varioustypes of input and output devices 103 108 may be supported via hardwareand software made part of the system 104 105, or by connecting adaptersfor those devices to built-in system connectors 104 such as USB ports,such as, for example, cable boxes, home appliances such as microwavesand washing machines, home automation systems such as those using X10,power wheelchairs, and other electronically- or electrically-controlledsystems.

OS Services Used to Abstract I/O Layer

The input and output (I/O) layer of hardware and software is abstractedby the system for modularity and ease of interfacing. The systempresents an API to provide access to relevant I/O activity with devicetype and communications protocol abstracted away, such as connection anddisconnection of devices, lists of connected devices, parameters ofconnected devices, data provided by devices, and writing data todevices. In the aspect here described, services of the Linux operatingsystem implement these features 109.

Main Application Uses Asynchronous Event Loop to Respond to VariousEvents

A main application 801 monitors I/O activity and delegates handlingthereof according to its configuration parameters, which may beconfigured via an API 803 that abstracts configuration parameters intohuman-understandable concepts. The application utilizes an asynchronous“event loop” 802 as shown in FIG. 9 to efficiently monitor connectionsand disconnections of devices 817 818, communicate with devices toreceive 808 and send 821 data and parameters, relay input data to thepipeline (thus triggering processing and output) 821, handle APIrequests (for example, requests to configure or re-configure the signalprocessing pipeline) 819, relay API responses and notifications 822 829,and handle timers. In the aspect here described, the application usesthe asynchronous event loop provided by the ‘libuv’ library of the Linuxoperating system.

Pipeline Represented as Graph, Nodes, Edges, Flow, Config API, Output Aspreviously described, the signal-processing pipeline is represented as anetwork graph 205 whose nodes 301 apply transformations to incoming dataas it becomes available, and whose edges 206 carry data from nodeoutputs to node inputs. In the aspect here described, the graph isimplemented in an object-oriented programming language (viz. Python) asa Graph object populated with Node objects and Edge objects, as furtherdescribed below.

The system presents a configuration API to enable the pipeline to beconfigured for a user's needs, which the system exposes to the user viaa GUI. Signal transforms are implemented as Node subclasses, and thiscapability is exposed to the user via the configuration API. Signalrouting is implemented via Edges that connect Node outputs to Nodeinputs, and this capability is also exposed to the user via theconfiguration API.

Nodes have input ports and output ports for receiving (reading) andsending (writing) data, and contain executable code that runs when datais available at an input port, typically applying some transformation tothe input data and writing the result to one or more output ports. Edgesspecify connections between an output port of a first Node and an inputport of a second Node; i.e., when data is written to the specifiedoutput port of the first Node it will be sent to the specified inputport of the second Node. Data from user interface devices enters thepipeline 205 via “source” nodes 302, passes through the graph asspecified by the edges 206 contained in the pipeline, being transformedby the intermediate nodes 301, and the transformed signals exit thepipeline via “sink” nodes 303 that pass data along to the output systembeing controlled or to an adaptor to an output system.

System Hardware (Single Board Computer)

In the aspect described, the hardware for the system is a commercialoff-the-shelf (COTS) single-board computer 905 (viz. Raspberry Pi 2.Model B) running a Linux operating system (viz. Raspbian) 906. In othercomputer-based aspects of the standalone system architecture, othercomputer boards and other operating systems may be used.

Main Application

Device Detection, Device Enumeration, Event Loop and Event Handling

In the aspect described, the system uses OS utilities such as Linux's“evdev” and “udev” for HID device enumeration and interaction. HIDdevices are uniquely defined by a vendor ID (VID) and product ID (PID),and the system assigns a slot to each device to handle duplicates of thesame product—the combination of VID, PID, and slot thus defines a uniqueID (DEVID) for a connected HID device. Using OS utilities, the eventloop 802 may monitor live HID device connections and disconnections 810812, and assign a DEVID to each currently connected device. The eventloop may use an OS utility to monitor signals from connected HIDdevices, abstracting the low-level USB HID protocol and exposing eachsignal datum to the system as an input “event” 808. These events areuniquely identified by an event type (ETYPE) and event code (ECODE), andthus the combination of DEVID, ETYPE and ECODE defines a unique event ID(EVENTID) for each input signal generated by a connected HID device.Each source node stores EVENTIDs for events it handles, and on eachinput event, the event loop checks to see if there is a source node inthe pipeline matching the EVENTID of the input event 820. If so, theevent loop injects the input event data into the pipeline via thematching source node 821; otherwise, the input event is ignored.

The system may support input sources other than USB HID devices (such asbutton-like inputs using 3.5 mm mono jacks, or wheelchair joysticksusing DB-9 connectors and custom protocols) by incorporating thehardware, electronics and software to handle communication with thedevice, and providing a similar unique addressing scheme for thatcategory of devices. The system may also implement deviceconnection/disconnection detection as appropriate for the type ofdevice.

Event Loop Overview

Once the main application is initialized, FIG. 9 illustrates theoperation of the event loop 802. On each cycle of the event loop, theapplication notes whether any relevant I/O event has occurred 805. Ifone or more data logging rules have been activated, the applicationchecks to see if the rule conditions are met 806, and if so logs dataaccordingly 807. If new user input data has been received 808, theapplication creates an “input datum received” 809 event for laterhandling. If an input or output devices has been connected 810 ordisconnected 812, the application creates a “device connected” 811 or“device disconnected” 813 event for later handling. If an API call hasbeen received 814, the application creates an “API call” 815 event forlater handling. The I/O monitoring portion of the application thenremains idle 830 until these generated events have been processed.

The application then processes generated events. If an “input datumreceived” event has occurred 816, the application checks to see if thereis a source node in the pipeline matching the input event 820. If so,the application injects the input event data into the pipeline via thematching source node 821, and sends an API notification to anyregistered configuration tool 822; otherwise, the input event isignored. If a “device connected event” has occurred 817, the applicationadds a listener callback to handle input events from that device 823,adds the device to a global device monitoring list 824, and sends an APInotification to any registered configuration tool 822. If a “devicedisconnected event” has occurred 818, the application removes itslistener callback for that device 825, removes the device to a globaldevice monitoring list 826, and sends an API notification to anyregistered configuration tool 822. If an “API call” event has occurred,the application takes the appropriate action instructed by the API call827, and if appropriate 828 sends an API response to any registeredconfiguration tool 829. When all generated events have been processed,this event processing portion of the application remains idle 830 untilnew I/O events have been handled by the I/O monitoring portion of theapplication.

API

At runtime, the event loop exposes an API (via a remote communicationsprotocol) for user configuration and monitoring. This supports pipelineediting and real-time view of device statuses and input data via agraphical user interface (GUI), which may run on the host OS, anothercomputer system connected on a local area network (LAN), or on acomputer system connected via a wide area network (WAN) such as theInternet. In addition to pipeline editing and monitoring, the API allowsfor editing of system settings such as logging levels and networkconfigurations. This architecture also enables user interaction with thesystem API directly via a script or a command line interface if desired,which can be convenient for remote debugging or development of advancedconfiguration and testing tools.

In this aspect, the TCP protocol is used to establish a connectionbetween the system and a user configuration tool, as well as to senddata bidirectionally between the system and the configuration tool. APIcommands are JSON data objects serialized as newline-delimited strings.When the system receives a string over TCP ending in a newlinecharacter, it deserializes the string as a data object and uses a lookuptable to execute an appropriate API response based on the data therein.The system similarly replies to API requests or pushes API notificationsas JSON data objects serialized as newline-delimited strings.

Pipeline Configuration

The API offers low-level modifications to the graph representation ofthe pipeline, such as addition, deletion, and querying of executablegraph nodes (Nodes) and connections between nodes (Edges), as well ashigher-level modifications (such as creating, configuring, or deletingcombinations of Nodes and Edges to implement certain desiredfunctionality, for example transforming an analog input signal foroutput to a digital output channel), and methods for configuring datalogging, data transmission, or certain operating parameters of thesystem (for example, network connectivity settings). The applicationrelays graph modification requests and queries to an API 204 provided bythe pipeline 203 that executes the requested modifications or queriesand returns messages as appropriate. The program constructs the pipelineat runtime via its API, and may do so from a saved configuration file orfrom live user editing. Graph modifications are atomic to allow for safemodification of the pipeline at runtime (i.e. while the pipeline is inoperation). The pipeline API can support disabling of some or all of thepipeline without modifying the pipeline graph by setting disable flagson certain nodes, for instance to suppress the output of sink nodesduring user modification so that the system does not send controlsignals to an output system during configuration.

Device Profiles, Querying and Configuration

While some devices are capable of providing some self-descriptiveinformation (e.g. via USB HID reports), it may be desirable for thesystem to have more complete device information than can be obtained byquerying a device. To this end, this aspect of the system supports“device profiles” containing “rich” descriptive device information, suchas user-friendly names for device inputs (e.g., “Right Trigger” insteadof “ABS RZ”), analog input types (setpoint: low/mid/high/none,spring-return: yes/no, etc.), or analog input ranges (e.g. to allowautomatic range scaling for control of a connected output system). Theseprofiles can be packaged with the system, generated by the user througha combination of auto-population and user editing, or shared forinstance via the Internet. When a device is plugged in, the systemchecks to see if it has an associated device profile. If so, it loadsdevice information from that profile; otherwise, it attempts to generatea profile based on the information it can query from the device, and ifconfigured to do so may query the user to provide improved or morecomplete information. The system can also support “device categoryprofiles” that match more than one type of HID device. For example, ifthe pipeline is configured to use input from a particular keyboard(e.g., Logitech K830 Illuminated Keyboard), it might not matter to theuser what model of keyboard is used (e.g., a Logitech G710+MechanicalGaming Keyboard could be an acceptable replacement). In that case it maybe inconvenient to have to update the specific keyboard expected by thepipeline if the user switches to another keyboard (or shares theirconfiguration with another user). In this case it might for instance bedesirable to assign pipeline input to a device category such as “USQWERTY Keyboard” that matches any keyboard with a standard QWERTYlayout, rather than a more specific device identifier like “LogitechK830 Illuminated Keyboard”. This aspect of the system such devicecategories, which can be implemented for instance using a set ofstandard HID event codes (e.g., codes corresponding to KEY_A, KEY_Z,KEY1, . . . , KEY0, etc.). In this case, if the system does not have anassociated device profile for a newly connected device (or if the systemhas been configured to check device category profiles before checkingdevice profiles), the system may then check to see if it has anassociated device category profile. Any device exposing that set ofevent codes is a match for the device category.

Pipeline

Graph Objects

As previously described, a signal processing pipeline is represented asa Graph object containing Node and Edge objects. The Graph objectmanages addition, removal, and querying of Node objects and Edgeobjects, as well as higher order combinations of these operations forconvenience. When an Edge is added, a reference to its target input portis added to the Node object containing the originating output port. Thisreference is called directly when data is available on that output port.When an Edge is removed, this reference is removed from the Node objectcontaining the originating output port.

Edge Objects

An Edge object defines a connection between an output port on a firstNode and an input port on a second Node. When data is available on thespecified output port of the first Node of an Edge object, the node willsend it to the specified input port of the second Node of that Edgeobject. It is valid for an output port to be connected via an Edge to aninput port on the same Node.

Node Objects

As shown in FIG. 10 , The base Node class defines a collection of one ormore named input ports and output ports 305 309, as well as an execute() function that can access data available 306 at the Node's input ports305, transform the data 307, and write data 308 310 to the Node's outputports 309. The Node maintains flags to indicate whether or not eachinput port contains unprocessed data. The execute( ) function is calledeach time data is written to an input port 304, and can use the inputflags to execute differently (or simply wait for more data) depending onwhich input ports have new data available 306; after the execute( )function completes, the Node returns to an idle state 311.

The Node may buffer data from input ports for later handling or fortime-based signal processing functions. The Node may accept structureddata containing named properties and data elements, typically used whenthe Node expects a set of related values to arrive at the same time. TheNode may store additional internal state to enable various signalprocessing and generation functions. The Node may have an ‘enable’ flagthat can be used to disable Node execution without removing it from aGraph. In addition to acting on data at input ports, Nodes may acceptinitialization parameters at object creation time that affect theirbehavior.

Each input port of a port array is accessible within the Node's execute() function via the name of the port array and the index of the inputport. The execute( ) function also has access to the length of the portarray; i.e., the number of inputs connected to the input port.

Sometimes it may be desirable for a Node to accept an unbounded numberof connections for a particular type of data input, as opposed to beinglimited to pre-specified named input ports. Therefore, as shown in FIG.11 , Nodes may support 312 named “port arrays” 313 that expand toaccommodate multiple connections (i.e. a possibly unbounded list ofconnections), along with base Node functionality including named inputports 314, an execute( ) function 315, and named output ports 316. Forexample, as shown in FIG. 12 , it may be desirable to create a summingnode 323 that applies a sum operator 324 to an arbitrary number ofinputs 318, which could naturally be represented as an input port.

Nodes may optionally perform static or dynamic type-checking. In theformer case, the node requires incoming connections to certify that theyoutput data of the type specified by the input port; otherwise, theconnection is disallowed in the Graph. In the latter case, the type ofthe incoming data is evaluated at runtime, and a program error is raisedif the data does not match the type specified by the input port.

Node Types

Several types of Nodes may help to implement the various functionalitydesired in the pipeline, some of which are shown in FIG. 12 and FIG. 13.

-   -   1. Transformation nodes 317 are the basic signal processing        elements of the pipeline. Transformation nodes perform some        signal processing function 315 on their input data 318 and write        the processed data to their output ports 319. Examples include a        stream combination node that combines output signals from        multiple nodes into a single stream, a logical summing node 332        that applies the logical sum operator 333 to inputs on an input        port, a signal thresholding node that applies a specified        threshold to an input signal, and a statistics node 327 that        performs a number of statistical computations on its current        inputs 318 such as mode 328, median 329, standard deviation 331,        and other statistical measures 330 such as quartiles and        variance, outputting each statistic to a different output port        319.    -   2. For time-dependent pipeline behavior, a Node can be        initialized with a timer object from the system event loop, and        can also check the OS clock as desired via a system call. The        timer can “wake” the node by calling its execute( ) function        after a given period of time has from when the timer was        previously started within the execute( ) function 307. The OS        clock can be used for instance to detect occurrence of        “double-taps” from a button-based input stream; if a second tap        signal arrives within a specified time following a first tap        signal as determined from the system clock, the Node might send        output to a “doubletap” output port. The timer object can be        used for instance to generate macro sequences; for example, the        node might generate a “key repeat” of a given repeat period, if        an input port is not deactivated before a given length of time,        by repeatedly calling the timer object to wake the node again        after that period has elapsed.    -   3. For time-based processing functions or other functions        involving state, a node can maintain state variables and buffer        a specified amount of historical signal data, and create output        signals using functions that can access and modify state and        buffer data. As a simple example, a history node 320 may save        input data to a buffer of a given size 321, shifting the buffer        contents to accommodate new data discarding old data when the        size is exceeded, and output the current history “slice” 322 to        an output port of the same size as the buffer. An alternate        implementation of a statistics node 327 may apply its        statistical computations to buffered data on each input as        opposed to applying its statistical computations to the current        inputs. An IIR filter node 334 for signal smoothing or gesture        detection (e.g. from trackpad input) could be implemented using        a data buffer 338 for timestamped 336 signal data 335, applying        an IIR filter 339 to the buffered data, and outputting the        filtered signal 337. A stateful keypress filter node 340 could        be implemented using a state 342 variable to track when a key is        in a “down” state and filtering out any further keypress signals        until after the key has returned to an “up” state 341.    -   4. Nodes that retain and process large amounts of signal data        may optionally be initialized with data buffers having speed-        and/or memory-efficient implementations (not necessarily shared        by other transformation nodes, as the buffers may add        unnecessary overhead in those cases). The buffer implementation        may also offer efficient numerical processing algorithms, such        as array operations, matrix multiplication, and convolutions,        e.g. using libraries like LAPACK and BLAS. These may be used for        instance to implement an IIR filter for signal smoothing, or for        gesture detection (e.g. from trackpad input). For example, in        the programming language Python, a fast buffer (as compared to        normal Python arrays) may be implemented using the Numpy        library, which contains a number of efficient numerical        processing functions and data structures. In one implementation,        the buffer is a circular buffer implemented using a Numpy array        of twice the desired buffer length (which we will call        “max_length”). As items are added the buffer, an index is        incremented until wrapping around at max_length. Each appended        item is added in two places in the array: (index % max_length)        and ((index % max_length)+max_length), where % is the modulus        operator. This way there is always a contiguous stretch of the        Numpy array representing the current values in the circular        buffer, which can be efficiently sliced for improved speed of        operations on the buffer data. The IIR filter 339 previously        described may be more efficiently implemented 343 using this        efficient ring buffer 344 instead of a default Python array 338.    -   5. It may be useful to allow one or more signal streams to be        overridden by other signal streams; this can be implemented with        a particular type of Transformation node we will call a Priority        node 325. In one implementation, this Priority node has an input        port array named “val”, expecting each incoming datum to have a        named “priority” attribute, and may have an output port “val”.        The Priority node stores the last datum of each “val” input, and        on each new datum, checks to see if any of the last saved data        has a greater priority 326. If so, no new output is written 326;        otherwise, the new datum is written 326 to the “val” output        port.    -   6. Source nodes have output ports but no input ports, and        generate signals either internally (e.g. using an event loop        timer object) or by an external call (for example in response to        user input from a user interface device). For instance, an HID        Source node may be used by the event loop to inject signals        generated by particular HID devices into the pipeline.    -   7. Sink nodes have input ports but no output ports, and are used        to implement “side effects” not affecting other nodes in the        Graph, such as sending output signals to a physical output        device or logging signals or pipeline execution.

Given the described node functionality and the capabilities available tosomeone skilled in the art of using an object-oriented programminglanguage (viz. Python), it is thus straightforward to implement thesignal transformations described previously (e.g., affine scaling,thresholding, hysteresis, arbitrary functions, signal limiting, functioncomposition, etc.) within the aspect described, and other signaltransformations, by creating Nodes with appropriate execute( ) functions(which may take full advantage of the features of an object-orientedprogramming language) and by taking advantage of the routing features ofthe Graph.

Graphical User Interface (GUI)

For convenience and ease of use, a GUI may expose API functions asinteractive or noninteractive static or dynamic graphical elements (suchas, for example, clickable and draggable elements). A concrete GUIexample is described below, but there are many possible GUIimplementations capable of interfacing with the system API in the spiritof this invention, any of which may constitute a useful extension of thesystem.

For instance, as shown in FIG. 15 , the GUI may use the API to retrieveand display 705 available channels 709 for output control. When a userclicks an output to control 707, the GUI uses the API to retrieve anddisplay a dropdown list 711 populated with currently connected inputdevices. When the user selects a device from this list 715, the GUI usesthe API to retrieve and display a list 713 of available input signalsthat can be generated from that input device. When the user selects aninput channel from that list, the GUI uses the API to generate an “inputstream” in the pipeline connecting the input channel to the outputchannel that was first clicked. An “input stream” is an abstraction fora sequence of nodes and edges beginning with a source node andoptionally ending with a sink node; this abstraction can make workingwith the pipeline graph more intuitive for human users. Depending on theselected input channel and output channel, the user may then bepresented with various settings for that input stream, such as setting athreshold on an analog signal, adding a deadband to the signal,inverting the signal, using the signal to trigger a specified macro,etc., and the GUI converts the user's selections to API commands thatmodify the pipeline accordingly. The GUI may also give visual indicatorsof both raw input signal values and pipeline output, so the user canplay with inputs to get a feel for how they work and get immediatefeedback about the effects of their stream settings. The GUI may alsoprovide options to save, load, delete, and duplicate configurations, sothe user can quickly change settings for different use cases or shareprofiles with other users. FIG. 15 shows an example GUI implementation,where the system is currently connected to a Logitech wheel, a joystick,a set of four digital input buttons, a mouse, a foot pedal, and a headarray as user interface devices, and is connected to an Xbox One gameconsole as an output system. A list of possible output channels formapping on the Xbox One (“X”, “Y”, “A”, “B”, “LS”, “RS”, “JX”, “JY”,“LB”, etc.) appears on the left side of the GUI 704. Unmapped channels,such as “LS” 706 only show the output channel label. Channels that havea mapping applied, such as “X” which is mapped to the “A” button on theLogitech wheel 705, also display a label indicating the applied mapping709. Here, the user has clicked the “B” output channel 707, whichgenerates an input stream to control that output channel, and brings upa list of clickable input channels 711 with labels 715 indicatingavailable input devices for mapping (“Logitech, “Joystick”, “Buttons”,“Mouse”, “Foot Pedal”, “Head Array”) 710. The user has then clicked“Buttons” 715 which brings up a list 713 of the possible input channels714 for mapping on the “Buttons” device (“Button 1”, “Button 2”, “Button3”, “Button 4”), and has clicked “Button 3” 716, which maps the “Button3” input channel on the “Buttons” device to the “B” input for the XboxOne system, and brings up a more detailed settings pane 717 for thatmapping. Channels that have already been mapped, such as “Button: Button1”, are greyed out 712. ‘X’ icons 708 are provided to unmap mappedchannels while editing or when hovering over the channels.

Architecture

In the aspect described, as shown in FIG. 8 , the GUI 703 is implementedas a “single page web application”. A web server 704 runs on the samecomputer as the main application, and serves an interactive web page onthe computer's “localhost” network, which can be accessed on the samecomputer using a web browser 702 such as Chromium. The web server isresponsible for dispatching and receiving API requests and responses 803to interact with the main application 801. In the aspect described, theAPI communication takes place over TCP with API requests and responsessent as JSON data serialized as new-line delimited strings.

In the aspect described, the served web application is implemented withJavascript, HTML, and CSS files, and communicates with the web serverusing WebSockets. User interactions trigger API requests andcorresponding responses, and API responses trigger changes in the UIdisplayed to the user.

With no changes to the main application API, the GUI may also beimplemented in a standalone browser such as that provided byelectron.io, thus combining the three GUI components 702 703 804 into asingle application. The GUI may alternatively be implemented as a nativeLinux application or as a native application on another OS. The GUI mayalternatively may be implemented on a different network-connected devicesuch as, for example, an iOS or Android phone.

Example

The following illustrates one non-limiting example of the system of thepresent disclosure for playing FIFA (a soccer video game) on the XboxOne (XB1) console with various input devices. When a user is playingFIFA on the XB1 using a normal XB1 controller without the use of thesystem described herein, a typical setup may include the followingcontrols mappings:

Overall:

Move the left thumbstick to control player movement

On offense:

-   -   Press ‘A’ to pass    -   Press ‘B’ to shoot    -   Press ‘Y’ to pass a through ball    -   Press ‘X’ to cross or perform a header        On defense:    -   Press ‘A’ to “contain”    -   Press ‘B’ to tackle or clear    -   Press ‘X’ to slide tackle

An example of an alternate gaming setup using the system of the presentdisclosure is described below, with the mapping shown in FIG. 14 .

Suppose the user connects a USB foot pedal (“PEDAL”), two 3.5 mm jackbuddy buttons (“BUTTON1”, “BUTTON2”), and a USB joystick (“JOYSTICK”) tothe system, and connects the system to an Xbox One console with FIFAloaded. Assume the pedal has a single digital channel (“PEDAL:PRESSED”)345, each button has a single digital channel (“BUTTON1:PRESSED” 347,“BUTTON2:PRESSED” 348), and the joystick includes, among other things,an analog channel for horizontal stick movement (“JOYSTICK: STICK_X”349), an analog channel for vertical stick movement (“JOYSTICK: STICK_Y”350), and a digital channel for a trigger on the joystick(“JOYSTICK:TRIGGER” 346). For the digital input channels, a value of “1”is received by the system when the user activates the respective input,“2” is repeatedly received while the user keeps the input activated, and“0” is received when the user releases the input. The XB1 digitalchannels (A, B, X, Y etc.) are “pressed” when sent a value of 1 and“unpressed” when sent a value of 0. The joystick analog channel valuesrange from 0 to 1023 (left to right, down to up), and the XB1 leftthumbstick channel values range from −32768 to 32767 (left to right,down to up).

Suppose the user wants to press the foot pedal to pass the ball, pullthe trigger to shoot, press the first buddy button to slide tackle,press the second buddy button to pass a through ball, and control playermovement using the USB joystick. Using a provided graphical interface,the user may “map” 351 352 353 354 355 356 the PEDAL:PRESSED 345 inputto the XB1 A button 357, JOYSTICK:TRIGGER 346 to the XB1 B button 358,BUTTON1:PRESSED 347 to the XB1 X button 359, BUTTON2:PRESSED 348 to theXB1 Y button 360, JOYSTICK:STICK_X 349 to the XB1 left thumbstick's Xaxis 361, and JOYSTICK:STICK_Y 350 to the XB1 left thumbstick's Y axis362. This may generate a pipeline 205 with the following configuration:

-   -   A source node for each digital input (PEDAL:PRESSED,        JOYSTICK:TRIGGER, BUTTON1:PRESSED, and BUTTON2:PRESSED) is each        connected to a respective processing node 351 352 353 354 that        outputs a “0” when receiving a “0”, “1” when receiving a 1, and        nothing otherwise (i.e. when receiving a 2 or any other input).        Each processing node is connected to a respective sink node 357        358 359 360 for each of the output channels XB1:A, XB1:B, XB1:X,        and XB1:Y.    -   Source nodes for the analog inputs JOYSTICK:STICK_X and        JOYSTICK:STICK_Y are each connected to a respective processing        node 355 356 that linearly scales input values from the range        [0, 1023] to the range [−32768, 32767] and outputs the scaled        value. The processing nodes are respectively connected to sink        nodes 361 362 for the output channels XB1:LSTICK_X and        XB1:LSTICK_Y.

The system may identify the specified input devices, interconvert theirsignals and feed the converted signals into the pipeline, executepipeline nodes, and interconvert the pipeline output and send theseconverted signals to the Xbox One.

1-10. (canceled)
 11. An apparatus for enabling a user to control one ormore output systems, where at least one aspect of the one or more outputsystems is desired to be controlled by a user action via a plurality ofinput devices and is not otherwise configured to be controlled by theuser action via the plurality of input devices, the apparatuscomprising: one or more input ports configured to receive input devicesignals conveying data pertaining to user actions from the plurality ofinput devices; a first interconversion logic configured to convert theinput device signals to converted input device signals having a dataformat compatible with a modular, reconfigurable signal processingpipeline; the modular, reconfigurable signal processing pipeline,configurable via a configuration API to route one or more of theconverted input device signals through at least one API-selected firstsignal transformation to create one or more transformed signals, thenroute the one or more transformed signals to one or more API-selectedaspects of the one or more output systems, thus creating one or morepre-output signals; a second interconversion logic configured to convertthe one or more pre-output signals to one or more output signals havinga data format suitable to control the one or more API-selected aspectsof the one or more output systems; one or more output ports configuredto transmit the one or more output signals to the one or more outputsystems; the configuration API; and a user interface for interactingwith the configuration API, wherein commands of the configuration APIare JSON data objects serialized as newline-delimited strings.